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ABSTRACT 

This consideration of validity issues in self-s tudy 
research argues that issues of validity in such research are first 
and foremost moral arguments about educational practice. The 
arguments point to narrative study because versions of narrative work 
are considered a necessary component of self-study. Narrative 
research has been criticized on positivist, analytical, literary, and 
post-structuralist grourds, but the appropriateness of self-study as 
research is justified by the fact that professionals are always 
obligated to monitor their own performance. Teacher educators 
involved in self-study are submitting their understanding of morally 
principled professional action to the scrutiny of peers. If a 
self-study piece is to be valid, then the educational merit of the 
practice under study must be seen to be appropriate. All other 
debates about the validity of self-study research should be 
subservient to debates on the educational appropriateness of the 
professional practice that is the object of the self-study. (Contains 
14 references.) (SLD) 
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Introduction 



My original intent, when proposing this paper, was to address problems of validity in the area of 
self-study research. At that time, I believed that self-study research was in need of this form of assessment 
because little attention had been accorded to validity during the relative youth of this style of research and 
because I was concerned that the substantial potential of self-study might be diminished were attention not 
given to validity. I had hoped, too, that I might build upon the papers presented at the self-study session 
at the 1994 AERA meeting. While I worked on this project, it became increasingly clear that the goals I 
had set for myself were too expansive. Not only is the range of research studies presented as self-studies 
last year large — narrative, interview, and other forms of qualitative study are represented, as is one 
quantitative study — but also, and fundamentally crippling to my plan, was my realization that issues of 
validity arc not restricted to self-study work but arc endemic to our research. Colleagues more able than I 
have worked at this area, so it seemed unlikely that I would bring my project to fruition in one paper. Yet 
as I worked through selected accounts of validity', it became clear that there was more to be said. This 
paper, then, is an initial attempt to mark out the territory and to note the landmarks that I could use. with 
your help, to clarify validity for our purposes. 

The plan for this paper is to first treat some sources of validity problems and to address them in 
turn in the initial two sections: traditional and technical accounts of validity', and arguments about the 
validity of narrative study (including the notion of falsifiability). These discussions point to the centrality 
of place and function in considering validity, so the next section of the paper argues for the place of self- 
study, and this leads to a discussion of the moral dimension of validity in an academic community of 
discourse. The overall argument of the paper, then, is that issues of validity in self study research are 
first and foremost moral arguments about educational practice. 

Before 1 begin, I should perhaps add caveats. First, some of my arguments point rather directly to 
narrative study because I believe that versions of narrative work arc a necessary component of self-study. 
Second, I tend to be rather loose in my definition of narrative, again with the intention of being inclusive 
rather than exclusive. 



1 Paper presented at the annual conference of the American Educational Research Association, San Francisco, April 
1995. The paper is from the research project "Case Study Research in Teachers* Professional Knowledge ' (Hugh 
Munby and Tom Russell, principal investigators), funded by the Social Sciences and 1 lumanities Research Council 
of Canada. Portions of this paper are derived from remarks prepared for a colloquium sponsored by the Department 
of Teaching and Teacher Education, College of Education, University of Arizona. 1 am grateful to colleagues at the 
University of Arizona for their comments. My attendance at the AERA conference is made possible, in part, by 
funding from the Office of Research Services of the School of Graduate Studies and Research. Queen's University. 
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Traditional and Technical Accounts of Validity 



The text 1 use in the introductory educational research course at Queen' s is the Shumacher and 
McMillan (1993) — now in its third edition. I favor this because it gives substantial attention (four 
chapters) to qualitative research, chapters I recommend to students in the second-level research course on 
qualitative methods. This text and others present validity in what I call a "technical" way. We are 
familiar with such statements as: "Internal reliability in a qualitative design addresses whether (or not), 
within a single study, multiple observers agree " "Could other researchers use the same data to generate 
the same explanations or constructs?" And we are equally at home with the idea of having our students 
"check ofT answers to questions like: "Is there good use of verbatim accounts of participant 
conversations, descriptions phrased as concretely and precisely as possible from field notes or recordings 
of observations, and direct quotations?" "Are there multiple researchers?" "Are participant research 
assistants used?" And "Are tape recorders, photographs, or videotapes used for data collection?" My 
experience with such questions is that students are at a loss to know how to arrive at a judgment of 
validity and reliability. Their check marks cannot be treated as interval data, and my well-intentioned 
suggestions about the responsibilities of thesis committee members and about the place of peer review 
offer little solace. Ultimately, the "technical" accounts are found wanting because something normative is 
missing or left unstated in one way or another. 

I need hardly add that some of the "technical" criteria are probably vexing for those who engage 
in self-study and narrative method. Indeed, satisfying the following might render the entire enterprise 
unworkable: "Did the researcher take into account the possible effect he or she might have on the setting 
in relation to the way the setting would be if there were no research done at all?" 2 

Happily, there are other accounts of validity and reliability', and my students typically consult the 
chapter by Eisenhart and Howe (1992) in the Handbook of Qualitative Research in Education. These 
authors offer their own approach to validity after considering three current approaches. What I have 
already mentioned is their "adaptations of the conventional approach." Their "alternatives to the 
conventional conception" abandons validity' and embraces trustworthiness, although I admit to being too 
dense to see much of a difference when it comes to making judgments; and their "eclecticism" is a rather 
weak version of the first two, and involves (again) making judgments about issues. Their own approach, 
"five general standards for validity," consists of the following, of which the first three are said to be "rules 
of thumb for systemic consideration of research studies qua arguments" (p. 657): 

1 . The fit between research questions, data collection procedures, and analysis techniques. 

2. The effective application of specific data collection and analysis techniques. 

3 . Alertness to and coherence of prior knowledge. 

4. Value constraints. 

5. Comprehensiveness. 

Within the fourth, 'Value constraints," Eisenhart and Howe discuss both internal and external 
valu^ arguments. Internal ones concern ethics and are not central to my argument at present. The 
external ones concern "whether (or not) the research is valuable for informing and improving educational 
practice" (p. 660) and include the charge that the research results and implications must be made 
understandable to the field of practice. I want to focus on their view that "valid studies must be 
worthwhile" (p. 660). First, and minor, Eisenhart and Howe do not actually make an argument for this 
assertion, although I think it relatively easy to fashion one. It would go something like this: research is 
an argument, arguments are tools, and the validity of a tool depends on showing that it does the job 
intended and that the job itself is worth doing in the first place. But this is a distraction. More basically, 1 
find the charge that validity' depends in part on the value of the research to be simultaneously attractive 
and problematic. It appeals because it underscores the importance oi the section called something like 



2 To my chagrin, I have lost the source of this marvelous question. Perhaps someone can provide me the missing 
citation. 



"Rationale," or "Significance of the Study " and I do my utmost to persuade students of the need to argue 
coherently at that place. But external value constraints as described by Eisenhart and Howe seem to 
require more. 

For a moment, invoking Toulmin's (1958) notion of the structure of argument helps to clarify 
my concern. His basic model shows that a conclusion (C) is developed from data (D) by \irtue of a 
warrant (W) that entitles us to move from one to the other. For us. because educational practice is 
normative, either the warrant or the conclusion must be normative, so the significance of a study must be 
grounded in an argued value position. Eisenhart and Howe (1992) appear to be acknowledging this when 
they state ' judgments of the worth of research project are difficult to make. They have the potential to be 
exceedingly biased" (p. 660). In this respect, I do not find the external value constraint standard of 
validity really gets me closer to answering questions about self-study and narrative because I see this 
standard as "significance of the study" with a slightly different spin on it. Neither are we told how such 
arguments could be fashioned nor what might constitute appropriate value premises. And my perplexity 
increases as I read on because, in the same paragraph, the authors refer the reader to the debate in the 
1988 Educational Researcher on whether or not educational research has "made an important 
contribution to educational practice." This is a future-referring condition which Komisar (1968) used 
with some devastation some years ago when conceptual analysis of the term "teaching* ' was at its peak. 

The future-referring condition is not helpful. Of course, I use it all the time, especially in 
coaching sabre. My suggestions might be accompanied by something like, "Trust me, it will make a 
difference," and this is frequently followed by incontrovertible evidence that I am again wrong. Although 
the justification of research might properly employ the future-referring condition, it is odd to see the same 
condition applied to establishing the validity of the research once done. In principle, then, the future- 
referring condition seems inappropriate to establishing the validity of any form of research. Additionally, 
I am not confident that self-study and narrative approaches want to wrap their validity too tightly with 
their arguments for significance. We are all too familiar with the nervous chuckle that follows the 
question, "How many more cases do we need?" which is inevitably posed by someone in my classes. We 
might ask of self-study, "How many more times do you need to study yourself?" and of narrative, "How 
many stories do w e need?" Questions like these should be answered but I am not yet ready for them. For 
now, it is enough to record that the traditional account's future-referring condition, as described by 
Eisenhart and Howe, is either inappropriate or insufficiently worked through. 



Arguments About the Validity of Narrative Study 

Although narrative method as developed by Connelly and Clandinin (e g, Connelly & Clandinin, 
1990), was primarily (I believe) a route to conceptualizing the character of teachers' professional 
knowledge (as storied), narrative method has assumed a life of its own, if you will, and is represented in 
some of what I take to be self-study research. I have encountered three types of criticism about narrative 
forms of research, which 1 will typify- here with reference to the work of Cizek ( 1 995), ODea ( 1 994). and 
Tochon(I994). 

The Critique on General Positivist Grounds 

To be fair, Cizck's paper is not really a critique neither is it directed exclusively toward narrative 
research. Rather, his is a complaint about the incursion (he calls it hegemony) of aspects of narrative 
research into quantitative research reports. Nevertheless, his points strike me as typifying social 
scientists* concerns for this research approach. For instance, he asks: 

If research doesn't relate to anything we currently know (i.e., theory -driven), if it doesn't 
address a question of interest posed by the researcher (i.e , hypothesis testing) or produce 
knowledge that others can use and is hound to a particular setting (i.e., not gcncralizablc) 
then how can it even be called research? (p. 27) 
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He goes on suggest, 'Qualitative researchers are often not so much practitioners as believers. It's almost 
a religious thing" (p. 27). And, in calling for assistance from philosophers, he wonders whatever 
happened to the concept of falsifiability, asking, 'is qualitative research falsifiable?" (p. 27). While I take 
it that Cizek intends to be somewhat playful, his piece nicely captures the difficulties of assuming that 
narrative (and 1 suppose self-study) research has a similar function to quantitative forms of research and 
should be bound by similar principles, like falsifiability. 

The Critique on Analytical and Literary Grounds 

There have been several philosophical critiques, and here I use O'Deas (1994) response to some 
of these as an illustration of the mode that they adopt. "Pursuing Truth in Narrative Research" (O'Dea, 
1994) is a recent attempt to address concerns about validity in narrative, such as Phillips' (1994) paper, 
which is written from an epistemological and analytical perspective. O'Dea's approach is from literary 
criticism, and she tends to set aside other positions: Connelly and Clandinin's view that narrative 
research goes beyond reliability, validity and generalizability, and that time, place, plot and scene are 
intended to work together to create the experiential quality of the narrative; van Manen's (1990) focus on 
features like animating and evocative description; and Barone's (1992) attachment to features like 
accessibility, compellingness. moral persuasion. O'Dea takes up the challenge in terms of artistic truth, 
building on Doris Lessing's "a writer must above all else speak the truth" and Margaret Murdoch's "Good 
art speaks truth, indeed *is' truth, perhaps the only truth." For O'Dea, artistic truth is grounded in 
authenticity, in being true to oneself. Not only does this involve taking account of what actually occurred, 
it also involves "existential freedom," the duty to question one's adherence to norms, roles, attitudes etc.. 
perpetuated by external society'. I have little difficulty' with this account of authenticity', but 1 become 
perplexed with her view about the place of narrative: 

The point of the process, however, is not for researchers and practitioners to entertain 
each other with provocative, compelling stories from classroom practice. Rather it is to 
encourage practitioners to reflect deeply and discerningly on their teaching practice, to 
see it from a variety of perspectives, to uncover and bring to conscious awareness the 
multiple levels of presuppositions that inform their perceptions and which determine 
(often unconsciously) their interpretations of particular situations, (p. 167) 

From this I take it that narrative is intended to be educational, and that is why 1 am perplexed. If 
narrative is meant to be educational, then it surely has an obligation to its readership beyond the 
obligations of literature: the truths need to be true, one would think, and clearly Phillips does. The 
literary truth espoused by O'Dea does not appear to help here for all she seems to have done is to replace 
the idea of truth with calls for authenticity', as in literature. For some literary theorists, literature itself 
does offer up truths, and we as readers understand that the features of plot, character and setting function 
to deliver the truth and that, in fulfilling this function, they do not themselves have to be accurate 
accounts of anything real, whatever that might mean, because literature is fiction. If this is so, then why 
should narrative inquirers be concerned for the validity' of a narrative's features, except perhaps to respond 
to Phillips and O'Dea? 3 ODea's concluding paragraph takes me into a rather different area of perplexity: 

And now finally one mr begin to understand Phillips* concern that essentially 
inaccurate stories may be used to further students' ''knowledge" of the exigencies of 
classroom practice. For if teachers* stories are indeed to be used as texts to guide the 
uninitiated, then they must be more than accessible, compelling and morally persuasive, 
they must offer more than animating, evocative descriptions of classroom events. In 
short, above all else, they must be true and reliable, they must render faithfully and 



1 Ail answer to this was given to me by Gary renstcrmacher during o recent discussion in his office. I le suggested 
that, in an effort to gain legitimacy, narrative inquiry might have appropriated the language of the academy's 
disciplines (science mid logic). If indeed narrative inquiry is following this route, pretending to be something which 
it is not, then it is being inauthentic and is behaving quite contrary to O'Dea's ideals. 

4 
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precisely the realities of classroom practice, and "compellingness," "animation," and so 
on, must serve only as a means towards that end. (p. 170) 

It is plain that ODea, and probably Phillips, have settled on a view of narrative inquiry as 
functional. I need to attend to this, so 1 have to set aside for now two other features of that paragraph: the 
extent to which prospective teachers are uninitiated, and the failure of O'Dea's argument to approach the 
question of how we, as a community, are to judge that the narrative is authentic (true and reliable). 

The Critique from Post-Structuralism 

For the third type of criticism of narrative (and so, I suppose, of self-study) I have selected 
Tochon' s (1994) trenchant critique. If I were to adopt a strict reading of narrative inquiry, I should be 
obliged to consider the theoretical elements of this paper in some detail. I will not pursue these here save 
to mention that Tochon draws substantially upon narratology and semiotics. His argument illustrates the 
use of semiotic tools (the semiotic square, actantial analysis, and focal analysis) as deconstructivist devices 
to ensure validity, and he uses the theoretical framework to establish the enormity of the risks of entering 
deeply into the field of personal analysis, of personal deconstruction. Tochon presents these risks in a 
table, "psychologizing and psycho-analyzing development, becoming dependent, uncritically adopting 
implicit ideological norms, submitting oneself to a conformity network, pathologizing professional 
problems, developing verbal rather than true identity, developing egotism and/or delusory experiences, 
imposing institutional confessions, taking the narrative tools for the life goals, justifying inaction, 
justifying gossip as a sort of criterial evaluation on its own, justifying salaries and social parasitism" (p. 
238). Tochon's concern is, "Narrative inquiry*, as it is usually known in teacher education, focuses on 
individual psychology, and may become narrative therapy" (p. 237). Tochon is doubtful that narrative 
inquirers are prepared or warranted to engage in such personal analysis, and he is concerned that without 
proper preparation, "the specificity of pedagogical intervention may be lost and replaced by a grand 
orientation keeping the real problem of the teacher's self-sufficiency out of sight" (p. 239). 



The Place of Self-study in Academic Life 

Tochon' s severe cautions may not fully apply to self-study — that's a matter for later investigation 
tvhich would require an examination of the work produced in this category' of research. Nevertheless, 
some of his cautions are certainly relevant to colleagues who begin to look carefully into the mirror. 
Central to these is Tochon's implicit view that narrative has an educational function. Y"A surprisingly, 
the notion of function is woven throughout the accounts of validity and the critical approaches I have 
considered above: function underlies the idea of a future-referring condition, function is clearly 
bothersome to Cizek (who translates it into hypothesis testing), and function is important to O'Dea's view 
that the literary features of narrative are to guide the uninitiated. The persistence of function in talk of 
validity cries out for attention, and in order to explore function and its connection to validity, it is 
necessary- to examine also " place" or appropriateness. At issue first, then, is the place of self-study in 
academic life, particularly in the life of teacher educators. 

One approach, but a short-sighted one, to considering the appropriateness of self-study as 
research, might come from recognizing that professionals are always obligated to monitor their own 
performance, because acting in the best interests of clients entails close inspection of what was once 
thought to be a best interest. Since this is part of being a professional, and not just part of being a 
professional academic, then activities like self-study seem not to warrant special status as research. 
Professional researchers, the argument might continue, are obligated as individuals to monitor their work 
by ensuring that it meets research standards, peer review being one but only one way by which this is 
done. On this argument, lawyers should inspect their work to find why they lose cases, and doctors should 
inspect theirs to find why they lose patients. One might be hard pressed to call this research. Rather, 
self-study might better belong in the category of "teaching " 



There are two principal problems with this line of reasoning: circumstantial and 
epistemological, but I do not mean that these are unconnected. The problem of circumstance arises from 
our past, when the education of teachers became the province of the universities. At .that time, teacher 
education professors, possibly to gain respectability, adopted not just the dominant meaning for research 
but also the dominant categorization of academic work: research, teaching and service. Ironically, we 
may have become victims of the success of our research, some of which has shown all too plainly that 
practical knowledge is different from propositional knowledge, that preparing teachers involves more than 
training them, and that teaching itself is hugely complex. In short, this research strongly illustrates that 
the work of professors af education should not be assumed to fall into the categories of work undertaken in 
other academic units. I think our scholarly activity needs a separate and unique identity which 
acknowledges an essential connection between what we do and what we profess, and which gives 
prominence to the moral character of our enterprise set, as it is, in the discourse of practice. And this 
leads directly to considering the epistemological problem of rejecting self-study as research; our 
knowledge and the communities it serves are fundamentally different from those of other sectors of the 
academy. 



The Educative Function of Self-study and Its Validity 

A clear account of the difference between these types of knowledge is advanced by Fenstermacher 
and Richardson (1994). Using psychology as their example, they distinguish between an educational 
psychology oriented toward the discourse of the discipline of psychology and an educational psychology 
oriented toward the discourse community of educators. 4 For Fenstermacher and Richardson, an essential 
feature of this difference is that the discourse community of educators is about practice and so has moral 
dimensions. Not surprisingly, a considerable portion of our research is directed at practice. Indeed, as 
Richardson (1994) has argued it is important to distinguish between two forms of research on 
professional practice: "practical inquiry undertaken by practitioners in improving their practice, and 
formal research undertaken by researchers or practitioners designed to contribute to an established ard 
general knowledge base*' (p. 5). It is apparent from this and from the nature of self-study, as undertaken 
by teacher educators, that self-study is intended fundamentally to be educative and so, when teacher 
educators engage in it, they are engaging in a normative activity just as they would be were they not 
studying their own practice. But self-study research involves more than professional practice, because it 
involves displaying one's practice to colleagues, Accordingly, self-study research becomes considerably 
more than sharing aspects of professional life, and more than giving teacher educators voice. By exposing 
their professional practice and its development to colleagues, those teacher educators involved in self- 
study are submitting their understanding of morally principled professional action to the scrutiny of peers 
who may be assumed to be better positioned to judge its normative features than teacher education 
students. On this count, self-study research appears to be a high-risk venture. 

It is clear to mc that the normative character of self-study provides its audience with an 
immediate entry into determining something of its validity. If a self-stucfy piece is to be valid then the 
educational merit of the practice under study must be seen to be appropriate. This is not future- 
referring, because the educational qualities of a professional practice are at issue here, not necessarily the 
consequences of the professional practice even though these may form part of the self-study's report. 
Neither am I suggesting that all that needs to be said of validity can be captured by the internal and 
external value constraints proposed by Eisenhart and Howe (1992). Their internal value constraints refer 
to the ethical conduct of research, and external value constraints "concern whether the research is 
valuable for informing and improving educational practice" (p. 660). Self-study is on educational 
practice, so the prior constraint is that the professional action is educational. To be sure, all educational 
research must meet ethical standards, should have some educational merit, and should be accessible. But 



ERIC 



1 In the same place. Fenstermacher and Richardson respond to Phillips* ( W4) concern for validity in narrative 
inquiry, and accuse him of confounding narrative itself, narrative inquiry, and narrative use. 
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the normative character of the stuff of self-study demands that initial questions about validity be directed 
at the educational values of the professional practice itself 

To my mind, all other debates about the validity of self-study research should be subservient to 
debates on the educational appropriateness of the professional practice that is the object of the self-study. 
Decisions about which professional activities in teacher education should be condoned if not fostered take 
precedence over how the research avoids Bacon's (1960) "Idols*' which "beset men's minds" (p. 47) and 
other traditional threats to validity. This strikes me as a useful point for beginning an inquiry into the 
validity of self-study research. 



References 

Bacon, F. (1960). The new organon and related writings (F. H. Anderson, Ed). New York: 
Bobbs-Merrill. (Original work published 1620) 

Barone, T. (1992). A narrative of enhanced professionalism: Educational researchers and 
popular storybooks about school people. Educational Researcher, 27(8), 15-24. 

Cizek, G. ( 1 995). Crunchy granola and the hegemony of the narrative. Educational Researcher, 
24(2\ 26-28. 

Connelly, F. M., & Clandinin. D. J., (1990). Stories of experience and narrative inquiry. 
Educational Researcher, 1 9(4), 2-14. 

Eisenhart, M., & Howe, K. (1992). Validity* in educational research. In M. LeCompte. W. 
Millroy, & J. Preissle, (Eds). Handbook of Qualitative Research in Education (pp. 643-680). San Diego, 
CA: Academic Press. 

Fenstermacher, G., & Richardson, V. (1994). Promoting confusion in educational psychology. 
How is it done? Educational Psychologist, 29(1), 49-55. 

Komisar, P. (1968). Teaching: Act and enterprise. In C. J. B. Macmillan & T. W. Nelson, 
(Eds). Concepts of teaching: Philosophical essays (pp. 63-88). Chicago, 1L: RandMcNally. 

CTDea. J. ((1994). Pursuing truth in narrative research. Journal of Philosophy of Education, 2#, 

161-171. 

Pliillips, D. (1994). Telling it straight: Issues in assessing narrative research. Educational 
Psychologist, 29{\) s 13-21. 

Richardson, V. (1994). Conducting research on practice. Educational Researcher, 23(5). 5-10. 

Shumacher, J , & McMillan, S. (1993). Research in education: A conceptual introduction. 
(3rded ). New York: HarperCollins. 

Tochon, F. (1994). Presence beyond the narrative: Scmiotic tools for deconstructing the personal 
story. Curriculum Studies, 2 X 22 1-247. 

Toulmin, S. (1958). The uses of argument. Cambridge: Cambridge University Press. 

van Manen, M. (1990). Researching lived experience. London, Ontario: Althousc Press. 



8 



7 



